S3Drive - File sorting and browsing

I have backed up my phone's media files, and I also uploaded files to completely different folders using rclone. I'm mentioning that just in case it matters. I can see those folders and files from S3Drive. When I use the app to browse Automatic Upload -> phone-model -> Camera, there's lots of files. But there's no way to scroll fast to the bottom, so scrolling manually takes ages. So I try to change the sort order to Name, Descending. At this point I'm always told that the cache is over an hour old. Regardless of whether I choose to regenerate the cache or not, I'm just shown an empty folder, as attached. How would I know if it's working/rebuilding still? I'm not sure the cache has ever been successfully rebuilt at this point, maybe that's the problem. Do I simply have too many files? At this point, if I browse back up to the root aka Files folder, that is listed as empty as well. But if I click on Albums at the bottom of the app, and then back over to Files, the folders are all shown again.

When I use the app to browse Automatic Upload -> phone-model -> Camera, there's lots of files. But there's no way to scroll fast to the bottom, so scrolling manually takes ages.

We will improve scrolling: https://s3drive.canny.io/feature-requests/p/implement-vertical-scroll-bar-for-folders-with-plenty-files

So I try to change the sort order to Name, Descending. At this point I'm always told that the cache is over an hour old. Regardless of whether I choose to regenerate the cache or not, I'm just shown an empty folder, as attached. How would I know if it's working/rebuilding still?

In a most recent release on Android there should be a relevant information in Logs if cache rebuild finishes or fail. There isn't a rebuild indicator currently, we will add one.

Do I simply have too many files?

Well, might be, speed depends on multiple factors, S3 provider and connection latency. We query S3 provider multiple times, in each request we get 1000 items, so if you have e.g. 20k files, we need to sequentially query S3 20 times which depending on S3 provider responsiveness and latency (usually physical distance between you and their server) it might take half a minute to 2-3 minutes. Do you approximately know how many files you have?

At this point, if I browse back up to the root aka Files folder, that is listed as empty as well. But if I click on Albums at the bottom of the app, and then back over to Files, the folders are all shown again.

This is now fixed in a most recent release. Thanks for your feedback.

Tom

When I use the app to browse Automatic Upload -> phone-model -> Camera, there's lots of files. But there's no way to scroll fast to the bottom, so scrolling manually takes ages.

We will improve scrolling: https://s3drive.canny.io/feature-requests/p/implement-vertical-scroll-bar-for-folders-with-plenty-files

So I try to change the sort order to Name, Descending. At this point I'm always told that the cache is over an hour old. Regardless of whether I choose to regenerate the cache or not, I'm just shown an empty folder, as attached. How would I know if it's working/rebuilding still?

In a most recent release on Android there should be a relevant information in Logs if cache rebuild finishes or fail. There isn't a rebuild indicator currently, we will add one.

Do I simply have too many files?

Well, might be, speed depends on multiple factors, S3 provider and connection latency. We query S3 provider multiple times, in each request we get 1000 items, so if you have e.g. 20k files, we need to sequentially query S3 20 times which depending on S3 provider responsiveness and latency (usually physical distance between you and their server) it might take half a minute to 2-3 minutes. Do you approximately know how many files you have?

At this point, if I browse back up to the root aka Files folder, that is listed as empty as well. But if I click on Albums at the bottom of the app, and then back over to Files, the folders are all shown again.

This is now fixed in a most recent release. Thanks for your feedback.

Do you approximately know how many files you have?

Probably 15-20k in the folder I'm trying to sort, and probably over 50k overall. I don't have the exact numbers. I'm still experiencing the same issues, by the way. If I try rebuilding the cache, I have no idea if anything is going on but I can wait 15-20 minutes and things still don't work. If I wait more than an hour, I'm again told the cache is more than 1 hour old and asked if I want to rebuild. Whether I reply yes or no, all content disappears from the app. I tried clearing the app cache on android, and I think around 70MB was cleared, but that doesn't seem to have made any difference. (edited)

In a 1.9.2 release we've added better indication when rebuild is going (there is a spinner) and if it fails for some reason there will be an error icon (with error as a tooltip)... also if rebuild was succesfull if you hover over (desktop) or long-tap (mobile) on the rebuild icon it will tell you how old is the cache. Please give it a chance and let me know if it's an improvement.

Thanks! I like it, it's easier to see what's going on. But still it fails after a little while, and much faster if I switch away from and back to the s3drive app. I get errors like this every time, but I'm on a stable 4G/5G network that rarely tends to fail. (edited)

5:03 PM

Could it be that my app loses network access when it's in the background?

Thanks for checking this. At the moment app can't execute this task in the background. Once you switch away from the app, after some short time (usually around ~xx seconds) the OS will pause the app. It doesn't necessarily explain the network issue, but might be related (we need to investigate that one). Well, it seems there are still couple improvements that we need to execute:

Implement background support,
Implement request retry in case of a network, back-end failure,
Implement process resume (if it was interrupted, then start from the last point),
Prevent app screen lock during rebuild, I am going add these to the feature list. Finally there is an existing feature item to implement diff rebuild. In other words if full inital sync was executed, technically it would be possible to use the Versioning API to get the latest change and update the existing dataset: https://s3drive.canny.io/feature-requests/p/implement-searchorder-index-diff-rebuild

(edited)

Changing to this setting resolved the issue for me immediately.

5:45 PM

1

That's cool, good to know. Perhaps we can temporarily solve it by updating our Docs with the recommended settings. Out of curiosity what was the previous / default setting? Was it Optimised or Restricted? Thanks!

The previous setting was Optimised, which I believe is the default setting.

For some reason, media backup has no folders selected any more. Perhaps they were lost when I cleared the app cache while trying to figure out this issue? The thing is, when I try to select some albums now under Media Backup-> Albums-> Select, the app seems to never get past the "thinking" stage, which looks like this: